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This paper reviews tourism studies published from 1996 to July 2009 that pertain to methodological 
approaches to website evaluation. The paper analyzes the initial work and continues up to recent 
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approaches: counting, automated, numerical computation, user judgment, and combined methods. The 
strengths and weaknesses of each method are examined. Research gaps and opportunities for future 
studies are discussed. 
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1. Introduction 

Since the launch of commercial Internet applications in the early 
1990s, many researchers have noted the potential of the World 
Wide Web in business, and advocated incorporating the Internet 
into the tourism industry (Burger, 1997; Clyde & Landfried, 1995). 
The rapid development of information technology in general and 
the Internet in particular has dramatically changed the tourism 
industry (Ho & Lee, 2007). It is widely accepted that the Internet can 
serve as an effective marketing tool in tourism (Buhalis, 2003; 
Buhalis & Law, 2008). It is a valuable tool for both suppliers and 
consumers for information dissemination, communication, and 
online purchasing. The rapid growth in the number of online users 
and the increasing rate of online transactions provide clear 
evidence of the popularity of the technology. Businesses, including 
customer-oriented and information-intensive tourism enterprises, 
are increasingly adopting e-business models to achieve their 
organizational goals. Maintaining an effective website has thus 
become vital for a business to strengthen its customer relationships 
and gain a larger market segment. 

Academic researchers have long advocated the importance of 
assessing website effectiveness. Lu and Yeung (1998), who were 
pioneers in the field, proposed a framework for evaluating website 
performance, in which the usefulness of a website is estimated 
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based on its functionality and usability. Evans and King (1999), 
Stern (2002), and Stout (1997) stated that website performance can 
be determined by network statistics such as hit rate and log anal¬ 
ysis. However, Fletcher, Poon, Pearce, and Comber (2002) noted 
that network statistics are subject to many limitations, including 
misleading and incomplete information, and Patton (2002) stated 
that practitioners should not analyze server log data. 

As a newly emerging research area, website evaluation has no 
globally accepted definition yet. However, the US Department of 
Health and Human Services (2006) broadly characterizes website 
evaluation as the act of determining a correct and comprehensive 
set of user requirements, ensuring that a website provides useful 
content that meets user expectations and setting usability goals. In 
general, prior studies on website evaluation fall into two broad 
categories: quantitative and qualitative. Quantitative studies 
usually generate performance indices or scores to capture the 
overall quality of a website. For instance, Faba-Perez, Guerrero- 
Bote, and de Moya-Anegon (2005) introduced a technique that 
compares web page measures such as text elements and link 
formatting. Suh, Lint, Hwang, and Kim (2004) used automated tools 
to analyze numerically measurable data, such as traffic-based and 
time-based data, on websites. Likewise, Cox and Dale (2002) built 
a scoring system with binary classifications for websites of various 
industries. Hardwick and Mackenzie (2003) applied three different 
scoring systems to evaluate 19 miscarriage-related websites. Lastly, 
Yeung and Lu (2004) conducted a longitudinal study of the func¬ 
tional characteristics of commercial websites in Hong Kong based 
on selected quantitative site attributes, and found that the websites 
were only marginally enhanced after 2.5 years. 
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In qualitative studies, the researchers assessed website quality 
without generating indices or scores. For instance, Heldal, Sjovold, 
and Heldal (2004) argued that the combination of branding, 
human-computer interaction, and usability could enhance website 
evaluation. Liang and Lai (2002) used a consumer-based approach 
to derive functional requirements for e-store design, and the 
empirical findings based on three e-bookstores showed that the 
quality of e-store design had a direct effect on the purchase decision 
making of students. Kim and Stoel (2004) used the WebQual scale 
to examine the dimensional hierarchy of apparel websites. The 
sample comprised female e-consumers, and the empirical results 
showed that the quality of websites selling apparel products could 
be conceptualized as a 12-dimension construct. 

In the context of tourism, Jang (2004) stated that online infor¬ 
mation search will become a major trend among travelers; with 
online reservations for travel products and services becoming an 
important application (Law & Hsu, 2006). Corigliano and Baggio 
(2006) argued that the quality and performance of commercial 
websites need to be monitored. In brief, website evaluation is of 
interest to academic researchers and industrial practitioners. Law 
and Bai (2006) found that published articles have presented various 
approaches and made efforts to improve the quality of commercial 
websites. Leung and Law (2006) reviewed information technology 
publications in leading tourism journals for the 1985-2004 period 
and found that networking was the most popular research area, 
with the highest growth rate. These studies demonstrate that travel 
website-related studies have become important to both industrial 
practitioners and academic researchers. 

Recently, review articles on tourism website evaluation have 
been published. Morrison, Taylor, and Douglas (2004) reviewed the 
application of the Balanced Scorecard (BSC) method in tourism 
website evaluation studies, and proposed a modified BSC method 
for future tourism and hospitality website evaluation. The authors 
also predicted that benchmarking will be a major approach in 
future research in this area. A benchmarking approach combines 
user perceptions with website performance to help owners identify 
the strengths and weaknesses of their own websites and in 
comparison with those of their competitors and the best practical 
examples in the industry. Hashim, Murphy, and Law (2007) focused 
on website design frameworks in their review of articles published 
from the 1990s to 2006. The authors extracted five dimensions of 
website quality based on the most researched online features of 
tourism and hospitality websites: information and process, value 
added, relationships, trust, and design and usability. They also 
found that the most popular attributes of hotel websites were 
reservations, contact information, promotions, and products and 
services. Similarly, Law and Bai (2006) reviewed published articles 
on website design and development in the Journal of Information 
Technology & Tourism and the proceedings of ENTER conferences 
(organized by the International Federation for Information Tech¬ 
nology and Travel & Tourism [IFITT]), which are the largest publi¬ 
cation channels on technology applications in travel and tourism. 
The authors found that the importance of website design is widely 
recognized, and that a number of website development evaluation 
instruments have appeared in the tourism field. Additionally, 
Buhalis and Law (2008) analyzed e-tourism related studies in the 
past 20 years and predicted the future of the e-tourism for next 10 
years. 

These studies give a basic overview of the historical develop¬ 
ment of website evaluation studies related to framework estab¬ 
lishment and evaluation. In terms of the coverage of the studies, 
Morrison et al. (2004) examined articles published mainly in 
tourism journals and conference proceedings up to 2003, Law and 
Bai (2006) analyzed only studies from two academic publications 
that are affiliated with IFITT, Hashim et al. (2007) examined the 


website attributes and dimensions identified in published articles 
on website evaluation. Lastly, the aim of Buhalis and Law’s (2008) 
study is to provide a general overview on Internet application to the 
tourism industry. 

This study attempts to provide an updated and comprehensive 
overview of prior tourism research that pertains to methodological 
approaches to website evaluation. Only the studies which investi¬ 
gated website assessment method are selected, and prior studies 
that are related to new website evaluation measurements or 
success factors are discussed. As Han and Mills (2006) commented 
that the use of the web in tourism marketing began in 1995, we 
review articles published from 1996 to early July in 2009. Murphy, 
Forrest, Wotring, and Brymer (1996) conducted a pioneer study 
that attempted to evaluate tourism and hospitality websites in an 
early stage of web development. 

2. Methodology 

In the present study, information on published articles was 
gathered from Science Direct (http://www.sciencedirect.com), 
EBSCOHost (http://search.ebscohost.com), and Google Scholar 
(http://scholar.google.com.hk), which are three of the largest and 
most popular online databases and search engines. The searching 
keywords included web site/website quality, web site/website eval¬ 
uation, web site/website assessment, and web site/website measure¬ 
ment. In addition, attempts were made to trace references cited in 
published articles. After careful screening of the articles, published 
studies were found that are directly pertained to the issue of 
website evaluation in the tourism and hospitality field. 

In total, 75 published articles were determined to be relevant to 
this study. Table 1 categorized the papers on the basis of industry 
sectors, regions, and evaluation instruments. The table also lists the 
number of published articles in each category on an annual basis. 
Apparently, in the category of industry sectors, the most popular one 
is for hospitality websites which were covered by 37 prior studies 
using different methods. Destination websites were the second most 
popular categories with 17 entries, and travel websites were the 
third most popular category with nine publications. In addition, 
Europe was the most investigated region with 24 papers, and Asia 
was the second most investigated region. As well, counting method 
was the most adopted instrument which was used by 25 prior 
studies with user judgment being the next with 11 publications. 

The next section provides an overview of these published arti¬ 
cles. The section following that identifies research gaps by 
comparing and contrasting these prior studies. The last section 
concludes the study and offers suggestions for future research. 

3. Findings 

Table 2 lists the related publications on tourism website evalu¬ 
ation studies. The list is in chronological order in terms of publi¬ 
cation year. The ten columns in Table 2 briefly describe each 
publication’s author/s and publication year, industrial sector, 
geographical region, research method, target sample of evaluation, 
instrument, evaluation/measurement, dependent variable/s, inde¬ 
pendent variable/s, and unit of analysis. 

In general, published articles covered 23 industrial sectors. These 
industrial sectors have been categorized into nine main categories 
included: i) hospitality websites that refer to hotel website (HWs), 
restaurant websites (RWs), lodging websites (LWs) ii) destination 
websites include regional tourism authority/organization websites 
(RTAs/RTOs), attraction websites (AWs), destination management/ 
marketing organization/system websites (DMOs/DMSs), national 
tourism organization websites (NTOs), rural destination websites 
(RDWs), attraction websites (AWs), visitor information center 
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Table 1 

Paper distribution by the industry sectors, regions and evaluation instruments. 


299 


Categories 


1996 

1998 

1999 

2000 

2001 

2002 

2003 

2004 

2005 

2006 

2007 

2008 

2009 

Total 

Industry sectors 

Hospitality websites 

2 


2 

3 

1 

6 

6 

4 

1 

6 

1 

3 

2 

37 


Destination websites 


1 

1 


1 

1 

2 

4 

2 

3 

2 

2 


19 


Travel supplier websites 




1 

1 


1 


1 

1 

1 

1 


7 


Airline websites 






2 


1 




1 


4 


Online travel guides 











1 



1 


Online travel magazines 











1 



1 


Travel websites 

1 





1 

1 



1 

4 


1 

9 


Travel search engine websites 










1 




1 


Travel blogs 











1 



1 

Regions 

Asia 






3 

5 

2 

1 

2 

2 

3 

2 

20 


North America 

2 



1 

1 

2 

1 

4 

2 

2 


1 

1 

17 


Europe 



2 

1 

1 

5 

2 

2 

1 

5 

1 

4 


24 


Africa 









1 





1 


Oceania 




1 


1 



1 

1 




4 


Worldwide 

1 

1 

1 

2 

1 


2 

1 

2 

2 

4 



17 

Evaluation Instrument 

Counting 

2 


3 

1 


5 

3 


2 

5 


3 

1 

25 


User judgment 




1 

2 

1 

1 

3 

1 


1 


1 

11 


Automated 






1 

1 



2 


2 


6 


Numerical computation 







2 




1 

1 

1 

5 


No actual evaluation 

1 

1 


2 



1 

1 


1 

3 



10 


Counting & User judgment 





1 

1 


1 


1 




4 


Counting & Automated 






2 


3 






5 


User judgment & Automated 







1 

1 



1 

1 


4 


Numerical computation & No actual evaluation 









1 

1 

1 



3 


Counting & Numerical computation 










1 




1 


websites (VICs), city tourism website (CTWs), official tourism web¬ 
site (OTWs), iii)Travel supplier websites which comprise travel 
agency websites (TAWs), brick-and-mortar travel retailer websites 
(BMTRs), tourism operator websites (TOWs), small and medium 
enterprise websites (SEMs), iv) airline websites (ALWs), v) travel 
search engine websites (TSEs), vi) travel blogs (TBs), vii) online travel 
guides (TGs), and viii) online travel magazines (TMs), and ix) travel 
websites (TWs). 

On the basis of the prior studies’ evaluation methods, five 
evaluation approaches were determined, including: i) counting (C), 
ii) user judgment (U), iii) automated (A), iv) numerical computation 
(N), and v) No actual evaluation (No). 

3.1. Coverage of industrial sectors and evaluation instruments 

These tourism website evaluation studies could be divided into 
two major categories: i) new evaluation model/instrument devel¬ 
opment, and ii) website evaluation using an adopted or modified 
model/instrument. The following subsection analyzes these cate¬ 
gories in detail. 

3.2. New evaluation model/instrument development 

In general, the new evaluation model/instrument development 
studies aimed to elaborate a new approach to travel-related web¬ 
site evaluation. Each of these approaches was usually developed for 
a specific industrial sector. For instance, Murphy, Forrest, and 
Wotring (1996) employed a qualitative method to collect basic 
functionality measurements from restaurant website suppliers, and 
these measurements were used to evaluate U.S. restaurant web¬ 
sites. Similarly, in their hotel website evaluation study, Murphy, 
Forrest, Wotring, and Brymer (1996) counted the features of hotel 
websites, and subsequently sought feedback from hotel managers. 

Some of these studies developed new evaluation frameworks 
but did not use them to evaluate actual websites. That is, the new 
frameworks were not applied or tested on publicly available 
websites. 


In general, information for website measurement and evalua¬ 
tion was collected by researchers in two ways. The first was to 
collect information or suggestions from consumers, suppliers, and 
researchers. For instance, van der Pijl, Haperen, Slikker, and Smits 
(1996) employed a qualitative research method to collect consumer 
perceptions of IT service and provided the results to suppliers, and 
to help suppliers think from the position of consumers. This 
research model measured consumer satisfaction with IT service, 
and divided service quality into two aspects of: i) client satisfaction 
and the theoretical perspective of quality, and ii) client satisfaction 
and the causal perspective of quality. Jung and Baker (1998) 
attempted to develop a framework to evaluate the market effec¬ 
tiveness of the World Wide Web in National Tourism Organizations 
(NTOs) from the perspective of suppliers. A questionnaire was used 
to collect the views of NTO suppliers, and a qualitative method was 
adopted to collect the views of researchers. The findings of this 
study indicated that most NTO professionals considered a website 
to be an effective marketing medium. Several factors that should be 
included in website design were identified, including ease of use, 
joy of use, content, interactivity, transaction support, added value, 
appearance, and clear navigation paths. Specifically, the study 
posited that website effectiveness can be measured by the number 
of hits, time spent, booking rate, interactivity, repeat visits, and 
feedback. However, Tierney (2000) argued that website effective¬ 
ness evaluation should go beyond counting the number of hits and 
page viewings. Tierney (2000) extended the study of Schonland 
and Williams (1996) and recommended that a Net Traveler Survey 
be used in future website evaluation studies. Buhalis and Spada 
(2000) developed a comprehensive list of success factors of desti¬ 
nation management systems (DMSs) by collecting the views of 
researchers, suppliers, and consumers. Additionally, Hu (2009) 
modified an electronic service quality (e-SQ) model and adopted 
a fuzzy multiple-criteria decision making model to determine a list 
of success factors to assess service quality of travel websites. 

Other studies reviewed the measurement methods described in 
the literature, and then categorized the collected attributes. For 
instance, Mills and Morrison (2003) developed an e-satisfaction 





Table 2 

Travel and tourism-related website evaluation studies from 1996 to July 2009. 


Author/s (year) Sector Region Methodology 


Murphy, Forrest, and RWs US 

Wotring (1996) 


Qualitative, 

Quantitative 


van der Pijl et al. (1996) TWs 


Worldwide Qualitative 


Murphy, Forrest, Wotring, HWs US 

and Brymer (1996) 


Quantitative, 

Qualitative 


Jung and Baker (1998) NTOs 


Worldwide Quantitative, 
Qualitative 


Gilbert etal. (1999) HWs 


Worldwide Quantitative, 
Qualitative 


Frew (1999) DMSs 


Morrison et al. (1999) HWs 


Austria; Qualitative, 

England; I Quantitative 

reland; 

Scotland 

UK Quantitative 


Tierney (2000) DWs US 


Quantitative 


Buhalis and Spada (2000) DMSs Worldw-ide Qualitative, 

Quantitative, 


Kaynama and Black (2000) TAWs Worldw-ide Quantitative, 

Qualitative 


Target Sample 


Researchers 


Consumers, 

Practitioners 


Consumers, 

Practitioners 


Researchers, 

Practitioners 


Practitioners 


Researchers 


Researchers 


Consumers 


Researchers, 

Suppliers, 

Consumers 


Practitioners 


Instrument Evaluation/ 

measurement 

Dependent 

Variable/s 

Independent 

Variable/s 

C 

Home page; Search- 
engine results; 
Classification; 

Website basic 
functions; Website 
communication 
functions and 
audiences 

Produced marketing 
strategies based 
on the performance 
of each factor 

Website performance 
factors 

No 

IT services; Client 
satisfaction and 
quality perspectives 

Quality of IT 
services 

Factors related to IT 
services 

C 

Home page; Search- 
engine results; 
Classification; Website 
basic functions; 

Website 
communication 
functions and 
audiences 

Promotion and 
marketing service, 
information, 
interactivity and 
technology, and 
management 

Website performance 
factors 

No 

Success factors of Web 
design: Internet value 
for marketing; website 
design factors 

Views about Internet 
value for marketing; 
measurement of 
percentage of using 
each method and rating 
factor of website design 

Internet value for 
marketing and factor 
of website design 

C 

RM (Relationship 
marketing) model 

Percentage of web 
features’ appearances 

Web features 

C 

Database, distribution, 
management, and 
operation 

Weighted score 
of DMS 

implementation 
characteristics and 
counting results 

DMS implementation 
characteristics 

c 

Technical, Marketing, 
Internal and customer 
critical success factors 

Evaluation results of 
website performance 

Website performance/ 
effectiveness features 
of a BSC (Balance 
scorecard) model 

No 

Test website 
effectiveness based on 
consumers’ point of 
view in Initial 
prophase; Post; Second 
Prophase 

Website effectiveness 

Percentage of initial 
prophase; Post; Second 
Prophase 

No 

Success criteria from 
consumers/tourism 
sectors/public sectors/ 
investors/tour 
operators/travel agents’ 
point of view 

Listed set of success 
criteria 

Success criteria 

U 

E-QUALITATIVE 

(e-SERVQUAL) 

Rating result of E-Qual 
factors 

Seven website service 
dimensions of E-Qual 


Unit of analysis 


Web content analysis 
from suppliers’ 
perspectives 


Gathering perspectives 
from both clients and 
practitioners on IT 
services 

Website content 
counting combined 
with practitioners’ 
perspectives on e- 
commerce 


Users’ views of using 
the Internet for NTOs 
with the importance 
and percentage of 
website design 

Counting web features, 
combining with 
interviewing hotel 
senior marketing 
directors 

Counting the 
appearances and rating 
the importance of the of 
DMS implementation 
characteristics 
Counting and 
evaluating website 
performance 

Using an online survey 
to gather consumers’ 
point of view on 
e-search 


Analyzing needs and 
wants of stakeholders 
by an integrated 
method 


Rating website service 
performance by users 
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Benckendorff and Black 
(2000) 

RTAs 

Australia 

Qualitative, 

Quantitative 

Practitioners 

C 

Oertel, Thio, and Feil (2001) 

DMSs 

Europe 

Quantitative 

Researchers 

C&U 

Jeong and Lambert (2001) 

LWs 

Worldwide 

Qualitative, 

Quantitative 

Consumers 

U 

Perdue (2001) 

Skires-orts 

US 

Qualitative 

Consumers 

U 

Blum and Fallon (2002) 

AWs 

Welsh 

Quantitative 

Consumers 

C 

Frey et al. (2002) 

HWs 

Switzerland 

Quantitative 

Laboratory Test 

C&A 

Wober et al. (2002) 

HWs 

Europe 

Quantitative, 

Qualitative 

Practitioners 

A 

Schegg et al. (2002) 

HWs 

Switzerland 

Quantitative 

Consumers 

C&A 

Lu et al. (2002) 

TWs 

China 

Quantitative 

Consumers 

U 

Jeong (2002) 

LWs 

US 

Quantitative 

Laboratory Test 

C 


Doolin et al. (2002) 

RTOs 

New Zealand 

Quantitative 

Laboratory Test 

C 

Wan (2002) 

HWs, TOWs 

Taiwan 

Quantitative, 

Laboratory, 

C&U 




Qualitative 

Researchers 



Planning, site 
management, site 
design, and site content 
Destination website in 
search engine, content, 
and functions 
Perceived usefulness, 
ease of use, 
accessibility, and 
attitude 

Consumer experience: 
Speed and quality of 
site accessibility: ease 
of navigation: Visual 
attractiveness of the 
site: Quality of 
Information content 
Product; Price; 
Promotion; Place; 
Customer relations; 
technical aspects 
Service processes, 
customer relationship, 
value added services, 
creation of trust, and 
cybermarketing 
Interactive features, 
navigational 
mechanisms, layout 
and multimedia 
characteristics, and 
content related features 
Service process, 
customer relationships, 
value creations, trust 
and cybermarketing 
Information content, 
ease of use, and 
functionality 
Purpose/value 
commercial website 
evaluation matrix 


Level 1-promotion; 

Level2-Provision 

LeveB-Processing 

Value-added model: 
user interface; Variety 
of Information and 
online reservation 


Element performance 


Different types of 
tourist behavior 

Intention to use 
information; use of 
information; and 
recommendation 
Overall resort quality as 
presented by the 
website 


Result of Marketspace 
model analysis 


Current use of Web 
technologies in the 
Swiss hotel industry 

Perceived ease of use 
and perceived 
usefulness 


Quality, technological 
standards, and 
website’s strategies 

Users’ perceptions and 
satisfactions on China 
tourism website 
Promotion, provision, 
and processing 


The utility of using 
interactivity to measure 
the relative maturity of 
tourism websites 
Ratings of websites of 
international tourist 
hotels and tour 
wholesalers 


Elements of successful 
website development 

Keyword search, 
functions, and content 
features 

Website information 
quality factors 


Website design factors 


Website content factors 
in Marketspace model 


Web marketing 
functions 


Website’s ease of use 
and usefulness 
attributes 


Over 200 Website 
criteria 


Functionality design 
features 

Four factors, including 
timely, customization, 
logistic, and sensational 


eMICA (extended 
Model of Internet 
Commerce Adoption) 

three criteria of user 
interface, variety of 
Information, and online 
reservation 


Content analysis 


External benchmarking 
analysis 

Website content 
analysis and weighted 
importance of website 
development factors 
Content analysis based 
on consumers’ 
experience 


Content analysis 


A benchmarking 
approach 


Web content 
extraction, content 
analysis and data 
mining 


Conjunction with 
personal observations 
benchmarked 

Content analysis based 
on users’ requirements 
and satisfaction 
Counting based on a list 
of purpose values for 
commercial website 
evaluation matrix, to 
seek the values that 
lodging websites are 
likely to provide to 
their customers 

Occurrence counting to 
evaluate website 
features 

Content analysis of the 
three criteria by rating 
the performance 

(continued on next page) 
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Table 2 ( continued ) 


Author/s (year) 

Sector 

Region 

Methodology 

Target Sample 

Instrument Evaluation/ 

measurement 

Dependent 

Variable/s 

Independent 

Variable/s 

Unit of analysis 

Law and Leung (2002) 

AWs 

Asia, North 
America 

Quantitative, 

Qualitative 

Laboratory Test, 
Researchers 

C 

An e-travel business 
framework: 

Information quality: 
System use; System 
quality: Service quality; 
Customer loyalty 

Result of reservation 
service quality 

Commercial website 
success factors 

Frequency counting to 
present website quality 

Aksu and Tarcan (2002) 

HWs 

Turkey 

Quantitative, 

Qualitative 

Researchers, 

Practitioners 

S 

Website address, 
purpose of, reservation, 
language, links, 
feedback 

Examination result 
and needs of website 
application for five-star 
hotels in Turkey 

Websites and Web 
applications of 
five-star hotels 
in Turkey 

Using counting method 
and interviews with 
managers to discover 
the needs of web 
applications in this 
industry 

Wober (2003) 

DWs 

Europe 

Quantitative 

Laboratory Test 

A 

Resource database and 
maintenance engine; 
Search engine; 

Protocol processing 
engine, and Website 
analyzer and 
management report 
engine 

Effectiveness of 
European city 
tourism web portal 

Success indicators for 
destination websites 

Web content and web 
usage mining by an 
automated website 
analytical tool 

Mills and Morrison (2003) 

TWs 

Worldwide 

Quantitative 

Laboratory Test 

No 

E-satisfaction model: 
TW interface; 

Perceived quality 
of TW Services and 
perceived value of TW; 
Customer online TW 
experience and 
Customer Satisfaction 
with TW 

Customer online TW 
experience and 
customer satisfaction 
with TWs 

Web interface, quality 
of TW services, and 
perceived value 
of TWs 

Developing a website 
evaluation model based 
on laboratory test but 
with no case study 

Huang and Law (2003) 

HWs 

China 

Qualitative, 

Quantitative 

Practitioners, 

Consumer 

N 

Product; Place; Price; 
Promotion; People; 
Packaging; 

Programming and 
Partnership; Customer; 
Needs & Wants; Cost to 
Customer; 

Convenience; 

Communication; 

Search Engines; 
Management Links; 
Useful Skills 

Overall Performance 
Score of 8Ps, 4Cs, and 
website promotion 

8Ps, 4Cs, and website 
promotion criterions 

Content analysis 

Feng et al. (2003) 

DMOs 

China and US 

Quantitative 

Laboratory Test 

C 

Modified BSC (Balanced 
scorecard) model: 
Marketing strategies; 
Web page design; 
Marketing information; 
Technical quality 

Websites’ actual 
performance 

Dimensions of a BSC 
model 

A benchmarking 
process to identify a set 
of critical success 
factors and evaluate 
hotel websites by 
content analysis 

Scharl et al. (2003) 

HWs 

Europe 

Quantitative 

Practitioners 

U & A 

Ease of use and website 
usefulness 

Web usage 

Criteria of a website’s 
easy of use and 
usefulness 

Web mining and 
a supplier survey on the 
effectiveness of hotel 
website 

Chung and Law (2003) 

HWs & TAWs 

Singapore 

Quantitative 

Practitioners 

C 

Basic information; E- 
commerce; 

Promotions; Secondary 
information; Services, 

Effectiveness of 
websites 

Consumers’ 
perceptions and 
website feature 
performance 

Gathering consumer 
perceptions of 
attributes together 
with website feature 


and technology analysis 
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Liang and Law (2003) HWs China Quantitative Consumers U 


Gilbert and Powell-Perry HWs Worldwide Quantitative Laboratory Test C 

(2003) 


Chung and Law (2003) HWs 


Hong Kong Quantitative Practitioners N 


Shchiglik and Barnes (2004) ALWs Worldwide Qualitative, Consumers LI 

Quantitative 


Aaberge et al. (2004) DWs 


Norway Qualitative Laboratory Test U 


Gupta et al. (2004) 

SEMs 

Welsh 

Quantitative 

Practitioners 

U & A 

Ham (2004) 

LWs 

US 

Quantitative 

Laboratory Test 

C& U 


So and Morrison (2004) NTOs 


East Asia 


Quantitative 


Laboratory Test C &A 


Facilities, customer 

Website functional 

Website functionality 

Gathering consumer 

contact, reservations, 
surrounding area and 
management of 
websites 

performance 

dimensions 

perceptions of 
attributes together 
with website feature 
analysis 

Relationship 
marketing model: 
information, 
reservation, loyalty 
program, newsletter, 
special gestures, 
feedback, customer 
service, public 
relations, value-added 
services, employee 
Website, channel 
member Website, and 
customized research. 

Hotel websites on 
the use of new 
technologies and 
multimedia 

Web marketing 
criterions 

Frequency counting 

Facilities, customer 
contact, reservations, 
surrounding area, and 
management of 
websites 

Website performance 

A conceptual 
framework of hotel 
website performance 

Content analysis 
combined with 
weighted established 
evaluation factors 

PAWQI (Perceived 

Perceived airline 

Four website quality 

Gathered consumers’ 

Airline Website 

Quality Instrument): 
site quality, 
information quality, 
interaction quality, and 
airline-specific quality 

website quality 

dimensions 

perceptions of airline 
website quality 

Technical functionality: 
The scope and 
correctness of the 
description of the 
object 

Consumers’ 
perceptions and 
websites’ actual 
performance 

Website content 
features 

Benchmarking process 

Access, information, 

Development of 

Design features and 

Content analysis with 

design, e-commerce/ 
immediacy and 
customer relationship 

customized CRM 
solutions 

e-commerce 

functionality, 

consumer satisfaction 

Impression; Content 
usefulness: Accuracy: 
Navigation; 
Accessibility; Online 
reservations; 

Timeliness of 
information 

Website effectiveness 

Evaluation criteria 
of limited-service 
of chain lodging 
operations 

Content analysis based 
on frequency counting 

Modified BSC(Balanced 

Website effectiveness 

Dimensions of 

A benchmarking 

scorecard) model: 
Technical; Marketing; 
Customer perspective; 
Destination 
information 
perspective 

evaluation from 

customers’ 

perspective 

modified BSC 
model 

approach 

(continued on next page ) 
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Table 2 ( continued ) 


Author/s (year) 

Sector 

Region 

Methodology 

Target Sample 

Kaplanidou and Vogt 
(2004) 

DMOs 

US 

Quantitative 

Consumers 

Douglas and Mills (2004) 

DMOs 

Caribbean 

Quantitative 

Laboratory Test 


Kline et al. (2004) HWs 


US Quantitative Laboratory Test 


Law, Ho, and Cheung (2004) HWs China and the Quantitative Consumers 

US 


Zhou and DeSantis (2005) CTWs 

North America, 
Europe, 

Asia, Australia 
and Africa 

Quantitative 

Researcher 

Law (2005) HWs 

World-wide 

Quantitative 

Consumers 


Mich et al. (2005) RDWs World-wide Qualitative Researchers 


Choi and Morrison (2005) BMTRs US Quantitative Laboratory Test 


Shi (2006) VICs 


Australia Quantitative Laboratory Test 


Han and Mills (2006) 


NTOs 


World-wide Quantitative 


Laboratory Test 


Instrument Evaluation/ 

measurement 

Dependent 

Variable/s 

Independent 

Variable/s 

Unit of analysis 

U 

Navigation, content, 
accessibility 
performance, and 
website usage and 
satisfaction 

Importance and Website performance 

performance of website factor 

factors 

Detect consumers’ 
perception on website 
importance and 
performance 

C & A 

Modified BSC Balanced 
scorecard) model: 
Technical aspects; User 
friendliness; Site 
attractiveness, and 
marketing effectiveness 

Website visitor 
retention 

Dimensions of 
a modified 

BSC model 

A benchmarking 
approach 

C & A 

Modified BSC 
(Balanced scorecard) 
model: User 
friendliness; Site 
attractiveness; 
Marketing 
effectiveness, and 
Technical aspects 

Strengths and 
Weaknesses of the B&B 
Websites 

Dimensions of 

Modified 

BSC model 

A benchmarking 
approach 

N 

Facilities, customer 
contact, reservations, 
surrounding area, and 
management of 
websites 

Website functional 
performance 

Website functionality 
Dimensions 

Content analysis from 
consumers perspective 
and comparison of the 
DMO website 
functionality 
performance between 
China and US 

C 

Website content and 
functions 

City tourism website 
usability 

The factors of website 
content and functions 

Content analysis based 
on counting method 

N & No 

No specific indication 
(hotel guests’ 
preferences and fuzzy 
assessments of website 
attributes) 

Hotel website 
performance 

Website attributes 

A fuzzy multi-criteria 
analysis model based 
on content analysis and 
user behavioral 
approach 

U 

Identity, content, 
services, location, 
maintenance, usability, 
and feasibility 

Website quality 

Website quality 
factors 

Content analysis based 
on users’ perspective 

C 

Modified BSC(Balanced 
scorecard) model: 
Technical; Customer; 
Marketing 
effectiveness; and 
travel agency 

Effectiveness of 
evaluated website 

Dimensions of 
a modified 

BSC model 

A benchmarking 
approach 

A 

Web content 
accessibility guidelines 
1.0 (WCAG) 

Web accessibility 

Attribute of web 
content accessibility 
guidelines 1.0 (WCAG) 

Content analysis by 
automatic method 

C&N 

An online promotion 
evaluation instrument: 
Aesthetics features; 
Informative features; 
and interactive features 

Marketing 
effectiveness 
of destination 
websites 

The factors of website 
promotion, and design 
features 

Benchmarking process 
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Park and Gretzel (2006) TSEs 


US 


Quantitative 


Laboratory Test No 


Bai et al. (2006) HWs Worldwide Quantitative Laboratory Test C 

Baloglu and Pekcan (2006) HWs Turkey Quantitative Laboratory Test C 

JarveRiinen (2006) TWs Europe Quantitative Laboratory Test C 

Chan and Law (2006) HWs Hong Kong Quantitative Laboratory Test A 

Zafiropoulos and Vrana HWs Greece Quantitative Practitioners, U&C 

(2006) Consumers 


Roney and Ozturan (2006) 

TAWs 

Turkey 

Quantitative 

Practitioners 

Essawy (2006) 

HWs 

UK 

Qualitative 

Consumers 

Beldona and Cai (2006) 

RDWs 

US 

Quantitative 

Laboratory Test 

Au Yeung and Law (2006) 

HWs 

HK 

Quantitative 

Researchers, 
Consumer 
Laboratory Test 


A novel research 
framework on search 
engine: Relative 
advantages, 
compatibility, 
complexity, trust, and 
subjective norm 
Operational Framework 
for Customer 
Relationship Marketing 
(Basic; Reactive; 
Accountable; Proactive; 
Partnership) 
Interactivity, 
navigation, and 
functionality 
Different website 
features for consumers 
in different experience 
levels 

An Automatic Website 
Evaluation System 
(AWES): website 
usability; Interface 
effectiveness; 
Information, Ease of 
navigation, and user 
friendliness 

A framework for the 
evaluation of hotel 
websites: Facilities, 
guest contact, 
reservation/price 
information, 
surrounding area, 
management of 
website, and company 
profile 

Information, before¬ 
sale and after-sale 
activities 

Interface, information, 
service, negatively 
affected the purchase 
and revisit intentions 

Content, interactivity 
and promotional value 
Language, layout and 
graphics, information 
architecture, user 
interface and 
navigation, and general. 


Individuals’ willingness 
to adopt and/or 
recommend travel 
search engines 


The attributes of 
catching travel search 
engine performance 


Mutual relationship 
(company type and 
size) 


e-RM 5 features 


Website effectiveness 
and marketing 
practices 

Features offered for 
inexperienced 
customers, features 
offered for more 
experienced customers 
The factors 

performance in design 
quality 


Website design 
characteristics 

Website usability 
features 


Website design features 


Hotel website overall 
performance 


Website function 
features 


Consumer attractions Information quality 


Purchase intention, 
revisit/ 

recommendation 


Usability dimension 


Website effectiveness Website quality factors 
Usability hazards index Usability dimensions 


Analysis on consumers’ 
perspective 


Content analysis and 
website classification 
counting 


Content analysis 


Comparison of website 
performance between 
experienced and 
inexperienced 
customers 
Using an automated 
system to evaluate 
website performance 


Hierarchical cluster 
analysis (HCA) 


Content analysis by 
frequency counting 

Protocol analysis 
methodology 


Content analysis 

Numerical calculation 
process based on 
consumer and 
professionals 
evaluation results 

(continued on next page ) 
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Table 2 ( continued ) 


Author/s (year) 

Sector 

Region 

Methodology 

Target Sample 

Instrument Evaluation/ 

measurement 

Dependent 

Variable/s 

Independent 

Variable/s 

Unit of analysis 

Law (2007) 

TWs 

Worldwide 

Quantitative 

Consumers 

N & No 

No specific indicators 

Consumers’ 
perceptions of quality 
of travel websites 

Multiple website 
attributes 

A fuzzy multiple- 
criteria decision¬ 
making model based on 
users’ assessment 

Choi et al. (2007) 

TWs, TBs, TGs, TMs 
OTWs 

Macau 

Qualitative, 

Quantitative 

Researchers 

A & U 

Narrative and visual 
information 

Keywords for each 
website sub-category, 
and performance of 
website sub-categories 

Most frequent ratings 
of words or phrases in 
rank order; attractions 
and visual information 
and website content 
and design factors 

Content analysis, 
ranking and counting 
percentages 

Lu et al. (2007) 

TWs 

China 

Quantitative 

Consumers 

N 

Website content and 
design 

Users’ perceptions and 
satisfactions 

Content and design 
features 

Content analysis and 
calculating by a fuzzy 
AHP model 

Stockdale and Borovicka 
(2007) 

RWs 

Austria 

Quantitative 

Laboratory Test 

U 

IS Success Model: 
information, system 
service quality 

Intention to use and 
users’ satisfaction 

Website quality 

Website evaluation 
based on users’ 
judgment 

Park et al. (2007) 

TAWs 

Worldwide 

Quantitative 

Laboratory Test 

No 

Ease of use, content, 
fulfillment, 
responsiveness and 
privacy, visual appeal, 
and security/privacy 

Willingness to use 
a website 

Website quality 

Factor analysis based 
on consumers 

experiences 

Ho and Lee (2007) 

TWs 

Worldwide 

Quantitative 

Consumers 

No 

information quality, 
security, website 
functionality, customer 
relationships, and 
responsiveness 

Purchasers’ 
perceptions/ 
expectations of e-travel 
service quality 

e-travel service quality 
scale factors 

Factor analysis based 
on consumers’ 
perceptions 

Park and Gretzel (2007) 

DWs 

Worldwide 

Qualitative 

Researchers 

No 

Ease of use; 

Information quality; 
visual appearance; 
Fulfillment; 
responsiveness 

Web evaluation; e- 
quality, e-satisfaction, 
and -loyalty 

Destination evaluation 
factors 

Categorization of 
review studies by meta 
analysis 

Qi, Leung, et al. (2008) 

HWs 

HK 

Quantitative 

Laboratory Test 

A 

An Automatic online 
Website Evaluation 
system 

Website content design 

HTML, image, external 
scripts, external CSS 
files, and multimedia 
files 

Evaluation of website 
performance by an 
automatic process 

Bauernfeind and Mitsche 
(2008) 

DMOs 

Europe 

Quantitative 

Researchers 

A&C 

Data envelopment 
analysis (DEA), 
linguistic offer, 
interactivity, and 
tourism content 

Website efficiency of 
tourism organizations 

Input and output 
Factors 

Benchmarking 
combining withData 
Envelopment Analysis 
(DEA) 

Hanai and Oguchi (2008) 

LWs 

Japan 

Quantitative 

Consumers 

C 

Surrounding area, 
transportation, 
building, service, 
payment options, price 
considerations, and 
facilities 

Information quality 

Lodging information on 
a website 

Correspondence 
analysis of lodging 
information 

Harison and Boonstra 
(2008) 

ALWs 

Holland 

Quantitative 

Researchers 

C 

Performance: Financial 
overview; Customer 
satisfaction; 
Traceability; 
Accessibility, Online 
sales process: Contact; 
Sales; After-sales; 
Customer support 

Website performance 
and consumer 
satisfaction 

Performance factors 
and online sale process 
indicators 

Content analysis, 
checking website 
performance on each 
indicators 
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Bevanda et al. (2008) 

Schmidt, Cantallops, and 
Santos (2008) 

Qi, Law, et al. (2008) 

Hu (2009) 


Xiong et al. (2009) 

Musante, Bojani, and Zhang 
(2009) 


TAWs Republic of Quantitative User based A 

Croatia 


HWs Spain & Brazil Quantitative Practitioners, C 

Researchers 


DMOs 


China Quantitative Researchers, N 

Consumers, 

Laboratory Test 


TWs Taiwan Quantitative User based No 


HWs & RWs U.S. Quantitative Researchers C 

HWs Singapore Quantitative Laboratory Test U 


Visual Appearance; Users’ behavior and 

Ease of Use; their requirements 

Fulfillment; 

Navigability; 

Accessibility; 

Personalization; 

Interactivity; 

Information quality 

Promotion, multimedia, Website performance 

navigability, customer 

retention, privacy and 

security, and service 

promptness 


Usability hazards index Usability dimensions 


Electronic service Website quality aspect 
quality (e-SQ) 


Language, layout and 
graphics, information 
architecture, user 
interface and 
navigation, and general 
Efficiency, Fulfillment; 
System availability; 
Security/privacy; 
Responsiveness; 
Compensation; 

Contact; Benefit; 
Customization/ 
personalization; 
Tangibility; Assurance/ 
trust; Continuous 
improvement; 

Website content 
accessibility Guidelines 

27Website content 
items 


Website accessibility 


Website attributes 
utilization and 
effectiveness for hotels 
of various class levels. 


Website design quality 
factors 


Website content and 
marketing factors 


Attributes within 
website content 
accessibility guidelines 
Content items 


Using an automated 
process to discover 
users’ perception 


Content analysis based 
on counting method 
.then adopted an 
exploratory factor 
analysis (EFA) to 
produce the final list of 
website characters 
Numerical calculation 
process based on 
consumers and 
professionals’ 
evaluation results 
A genetic-algorithm- 
based method fuzzy 
multiple-criteria 
decision making 
(MCDM) problem 
model 


Content analysis by 
frequency counting 

Rating content items 
for each selected 
website 
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model for travel website evaluation, which consists of three main 
dimensions: interface, perceived quality, and value. Park and 
Gretzel (2006) developed a framework for travel search engine 
evaluation. Introducing innovation diffusion theory (IDT), which 
was originally proposed by Rogers (1995), and comparing it with 
the technology acceptance model (TAM), Park and Gretzel (2006) 
established a model that comprises five measurements: relative 
advantage, compatibility, complexity, trust, and subjective norm. 
Ho and Lee (2007) conducted a comprehensive review and 
collected a list of attributes for measuring e-travel service. These 
attributes were then grouped into five dimensions: information 
quality, security, website functionality, customer relationships, and 
responsiveness. Park, Gretzel, and Sirakaya-Turk (2007) used 
a similar approach to create a comprehensive list of attributes for 
evaluating travel agency websites. Moreover, Park and Gretzel 
(2007) published a review study focused on analyzing both tourism 
and IT related studies, producing a comprehensive list of success 
factors for destination marketing websites. 

3.3. Website evaluation using an adopted or modified model/ 
instrument 

Prior studies that adopted or modified existing models/evalua¬ 
tion instruments to evaluate selected websites are grouped into this 
category. For example, Frew (1999) compiled a list of success factors 
in England, Ireland, and Scotland’s DMSs, and a panel of experts 
was invited to indicate the importance of these factors. Frew (1999) 
then determined which of these factors existed on selected web¬ 
sites, and the score of importance and existence of the dimensions 
reflected the overall performance of the evaluated DMSs. Kaynama 
and Black (2000) modified Parasuraman, Zeithaml, and Berry 
(1985) service quality (SERVQUAL) model, and produced a new 
model called E-QUAL to measure online travel service quality. The 
overall performance of the selected websites measured using E- 
QUAL was based on the level of consumer satisfaction. 

Prior studies in this category can be divided into five subgroups 
based on their evaluation method: counting, user judgment, 
automated, numerical computation, and combined method. 

3.33. Counting methods 

A counting method is used to evaluate a website’s performance 
or to determine its content richness. This type of evaluation method 
has two requirements. First, a well-prepared checklist is required to 
verify the existence of attributes on a website. In general, most 
items on a checklist are taken from adopted or modified models. 
Second, a group of people is needed to do the actual counting in 
a laboratory. Depending on the nature of the study, the website 
assessors can be consumers, suppliers, practitioners, policymakers, 
researchers, or students. 

Published articles provide the results of counting specific 
features of website content, together with the views of consumers 
and/or practitioners. For instance, Murphy, Forrest, Wotring, and 
Brymer (1996) applied a counting method to evaluate hotel web¬ 
site features and produced a set of evaluation factors. Chiang 
(2003) evaluated the effectiveness of business-to-business (B2B) 
online marketing in Singapore’s hotel industry using Murphy, 
Forrest, Wotring, and Brymer (1996) research approach. Consumer 
expectations were then identified using a structured questionnaire 
with nine assessment factors covering basic information, e- 
commerce, promotions, secondary information, services, and 
technology on 26 hotel websites. Frew (1999) established a list of 
success factors for DMSs and used the list to evaluate destination 
websites in several European countries. The author then combined 
the attribute importance ratings to generate a score for each 
website. 


For their evaluation of destination websites, Benckendorff and 
Black (2000) conducted content analysis of data collected from the 
Australian Regional Tourism Authorities (RTAs) to assess four major 
dimensions - planning, site management, site design, and site 
content - of the selected websites. On the basis of Benckendorff and 
Black’s (2000) work, Baloglu and Pekcan (2006) conducted a web¬ 
site evaluation study that analyzed website content of a group of 4- 
and 5-star hotels in Turkey in terms of design characteristics and 
Internet marketing practices. Doolin, Burgess, and Cooper (2002) 
presented an extended model of Internet Commerce Adoption 
(eMICA) based on a benchmarking process and counting method to 
evaluate the level of website development in New Zealand’s 
regional tourism organizations. The eMICA model has three distinct 
stages that incorporate three levels of the business process: web- 
based promotion, provision of information and services, and 
transaction processing. 

Morrison, Taylor, Morrison, and Morrison (1999) applied the 
Balanced Scorecard (BSC) approach in the tourism context to 
evaluate 16 hotel websites in Scotland. The model includes multiple 
critical success factors based on four perspectives: technical, 
marketing, internal critical, and customer critical. The researchers 
also marked website performance based on the error rate and three 
levels of download speed. Each hotel website received a total score 
that represented the site’s performance. This was the first tourism 
study to use the BSC in website evaluation. The approach was then 
modified by different researchers to match the specific needs of 
different industrial sectors or geographical regions. For example, 
Feng, Morrison, and Ismail (2003) evaluated destination marketing 
organization (DMO) websites in China and the U.S., and Choi and 
Morrison (2005) evaluated brick-and-mortar travel retailer web¬ 
sites in the U. Douglas and Mills (2004); Kline, Morrison, and John 
(2004) and So and Morrison (2004) also used modified BSC models 
in their studies. The latter studies, however, adopted automated 
instruments to assess the technical aspect. 

Gilbert, Powell-Perry, and Widijoso (1999) evaluated the service 
quality of 143 hotel websites based on a modified relationship 
marketing (RM) model. Gilbert and Powell-Perry (2003) then 
applied the RM model to website facility testing and investigated 
the use of multimedia on hotel websites. Bai, Hu, and Jang (2006) 
modified the RM model for use as a strategic marketing model for 
hotel website feature analysis. 

Some studies have evaluated websites using only the counting 
method. For instance, Zhou and DeSantis (2005) assessed the 
content of city tourism websites of North America, Europe, Asia, 
Australia and Africa. Law and Leung (2002) modified Liu and Arnett’s 
(2000) model and analyzed the content of Asian and North American 
airfare websites. The empirical findings indicated that the latter 
group of sites provided more features than did their Asian coun¬ 
terparts. Similarly, Blum and Fallon (2002) assessed 53 Welsh visitor 
attraction websites using a checklist that was produced based on the 
Marketspace model, which was originally developed by Dutta, 
Kwan, and Segev (1998). The model includes six main factors: 
product, price, promotion, place, customer relations, and technical 
aspects. All of these features were counted for the website analysis. 
Jeong (2002) adopted Ho’s (1997) value/purpose matrix to evaluate 
35 lodging websites in the United States. This approach includes 
counting features and comparing their values, and then analyzing 
the quality of these features. The empirical findings indicated that 
very few of the lodging websites motivated potential customers to 
make online purchases. Also, Xiong, Cobanoglu, Cummings, and 
DeMicco (2009) evaluated the accessibility of US hotel and restau¬ 
rants websites by Website Content Accessibility Guidelines 1.0 
(WCAG). Research findings indicated that the overall accessibility of 
the US hospitality websites was low and the primary problem was 
a failure in providing alternative text for non-text elements. 
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3.3.2. User judgment methods 

Studies adopting user judgment methods evaluated user satis¬ 
faction or perceptions. Depending on the nature of a study, users 
were any combinations of academic researchers, industrial practi¬ 
tioners, policymakers, and consumers. However, most of the 
studies in this group chose consumers, including potential 
consumers, as evaluators to measure different aspects and levels of 
user satisfaction, and only a few involved practitioners in the 
evaluation process. 

Jeong and Lambert (2001) tested a framework that evaluated the 
information quality of lodging websites based on consumer 
perceptions. Their model comprised four measures of information 
quality: perceived usefulness, perceived ease of use, perceived 
accessibility, and attitude. Perdue (2001) presented a conceptual 
model for evaluating North American ski resort websites, which 
assessed overall website quality, the visual attractiveness of the 
site, its information content, and existing resort images. In addition, 
Lu, Lu, and Zhang (2002) evaluated information content, ease of 
use, and functionality of China tourism websites from the Chinese 
users’ perspectives. Mich, Franch, and Martini (2005) developed 
a modular approach called the “quality model factory”, which 
extends the seven-loci meta-model, to evaluate unique destination 
websites. Shchiglik and Barnes (2004) developed a perceived 
airline website quality instrument (PAWQI) to evaluate websites 
from the perspective of consumers using a benchmarking process. 
Zafiropoulos and Vrana (2006) used a benchmarking process to 
evaluate hotel websites. Their model employed hierarchical cluster 
analysis (HCA) to classify the selected attributes. Extending the 
website usability literature, Essawy (2006) evaluated U.K. hotel 
websites using a protocol analysis method. The researcher con¬ 
ducted three four-person discussion groups that focused on web¬ 
site usability performance and further hotel website development. 
In a similar study, Stockdale and Borovicka (2007) developed an 
easy-to-use website evaluation instrument for travel and hospi¬ 
tality-related website evaluation, and conducted a pilot study using 
restaurant websites. This instrument originated from an existing 
information system e-commerce success model. 

Aaberge, Grotte, Haugen, Skogseid, and 01nes (2004); Liang and 
Law (2003) and Lu, Deng, and Wang (2007) used similar 
approaches to evaluate website performance on the basis of 
consumer satisfaction. Kaynama and Black (2000) modified the 
SERVQUAL model and developed a new instrument, E-QUAL, to 
measure the quality of website service by collecting data from 
industrial practitioners. 

3.3.3. Automated methods 

Automated methods involve the evaluation of websites using 
software systems. Researchers have developed, based on their 
research purpose, different kinds of website evaluation software. 
The advantages of an automated method include consistency in 
evaluation and a relatively faster process, compared to human- 
based evaluation. For instance, Wober (2003) and Wober, Scharl, 
Natter, and Taudes (2002) used content mining and Web usage 
mining tools for hotel and inter-regional tourism portal develop¬ 
ment. Similarly, Bauernfeind and Mitsche (2008) applied data 
envelopment analysis (DEA) to evaluate destination website 
quality. Shi (2006) evaluated the accessibility of visitor information 
centers (VICs) using a freely available online tool, the Bobby online 
free portal (Watchfire, 2004). Chan and Law (2006) and Qi, Leung, 
Law, and Buhalis (2008) adopted different online evaluation 
systems to evaluate the quality of Hong Kong hotel websites. 
Moreover, Bevanda, Grzinic, and Cervar (2008) adopted data 
mining methods that detect consumers' perceptions and require¬ 
ments for travel agent websites in a automated way. Specifically, 
the research evaluated website visual appearance, ease of use, 


fulfillment, navigability, accessibility, personalization, interactivity, 
and information quality. 

3.3.4. Numerical computation 

Numerical computation methods use mathematical functions to 
compute tourism website performance based on a number of 
aspects. Performance is usually represented by a set of numeric 
scores. Chung and Law (2003) developed five factors for evaluating 
the functionality performance of Hong Kong hotel websites. Overall 
site performance was calculated using a mathematical approach 
that combined the importance of each attribute based on user 
input. Huang and Law (2003) applied a statistical approach to 
mainland Chinese hotel website evaluation, which was modified 
from that of Chung and Law (2003) by integrating the importance 
of attributes and website performance. Au Yeung and Law (2006) 
evaluated the usability performance of 77 Hong Kong hotel web¬ 
sites using a heuristic algorithm for the optimization of mathematic 
functions by incorporating the perceived importance of different 
attributes using a sample of consumers, hotel managers, and IT 
consultants. The findings showed a significant difference between 
the websites of chain and independent hotels. Qi, Law, and Buhalis 
(2008) applied the approach but with a modified evaluation 
instrument to evaluate China’s DMO websites. In addition, Law 
(2005, 2007) presented two approaches that used different fuzzy 
mathematical models for the evaluation of travel and hotel web¬ 
sites. Lu et al. (2007) combined the analytical hierarchy process 
(AHP) and fuzzy synthetic evaluation (FSE) to assess the website 
effectiveness of travel networks in China. 

3.3.5. Combined methods 

Researchers have also used different combinations of website 
evaluation methods. For example, Frey, Schegg, and Steiner (2002) 
evaluated hotel websites in Switzerland assisted by a software 
system that generated search reports, which contained counts for 
each search criterion, a list of all sub-links visited, the location of 
dead links, and global statistical data about the websites. Douglas 
and Mills (2004); Schegg, Steiner, Frey, and Murphy (2002); So and 
Morrison (2004) and Kline et al. (2004) adopted automated 
methods to count specific features of websites. 

Some studies also combined user judgment with automated 
methods. For example, Gupta, Jones, and Coleman (2004) evaluated 
the performance of Welsh small to medium enterprise (SME) 
websites based on the perspective of suppliers, and download 
speed was tested using online software. Scharl, Wober, and Bauer 
(2003) also applied this approach in their study of European hotel 
websites. Choi, Lehto, and Morrison (2007) used a data mining 
technique for website multimedia quality testing, and the perfor¬ 
mance evaluation was then combined with expert input. Han and 
Mills (2006) examined the marketing effectiveness of hospitality 
and tourism websites by combining counting and numerical 
computation methods into one benchmarked instrument. The 
numerical computation method adopted in their study followed 
the statistical technique of the Kruskal-Wallis test. 

4. Research gaps 

Fig. 1 summarizes the methodological approaches used in prior 
research on website evaluation in tourism and hospitality. 

Five evaluation approaches, namely, counting, automated, 
numerical computation, user judgment, and combined methods, 
were adopted by tourism researchers. Each of these approaches has 
merits and limitations. For instance, a counting method can identify 
the existence of certain website features. These features, however, 
may not indicate the ease of use of the site. Also, it can be difficult to 
ask users to measure the download speed and file size of a website. 
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Fig. 1. Methodological approaches for the evaluation of tourism websites. 


Hence, many researchers applied user judgment and automated 
methods in their studies. However, a user judgment method 
examines user satisfaction and may not provide a clear picture of 
overall website performance in numeric form; while an automated 
approach is useful for testing the technical performance of certain 
features of a website rather than the site’s overall performance. In 
addition, an automated approach neglects user involvement. A 
numerical computation method appears to be a more advanced 
approach, using a mathematical computation process to produce 
numeric scores for performance evaluation. However, the weak¬ 
ness of this approach is the complicated process involved, which 
can be difficult for users to comprehend. A combined approach 
brings together the advantages of these different approaches. Such 
an approach, however, can still be modified in terms of sophisti¬ 
cation levels. 

Most tourism website evaluation research focuses on measuring 
website information and process, value added, relationship, trust 
and design, and usability aspects (Hashim et al., 2007). User 
involvement becomes especially important when measuring 
consumer service satisfaction and the perceived importance of 
attributes (Stockdale & Borovicka, 2007). It is, therefore, essential 
for future website evaluation research to keep refining the existing 
approaches rather than simply applying the existing approaches to 
different datasets. The adoption of a combination of methods 
provides a range of results that can satisfy the different needs of the 
entire range of stakeholders. 

However, as noted by Morrison et al. (2004), it is difficult to 
unite all instruments to form a single standard for website evalu¬ 
ation. Still, the numerical computation approach appears to be 
promising as it is measurable, repeatable, and can likely be used to 
form a long-term strategy. However, the limitations, such as 
comprehension issues, that are associated with this approach need 
to be addressed before the approach can be more widely employed. 
In short, there are no absolutely right or wrong methodological 
approaches, nor a standard approach for all sectors in the industry. 
Researchers should choose the evaluation approach that is most 
appropriate for their specific research objectives, target markets, 
and stakeholders. In summary, the major challenges in tourism 
website evaluation are: i) the existing techniques are largely 
adopted from other disciplines and a truly tourism-oriented 


evaluation technique has yet to appear, and ii) methods involving 
human subjects are subject to personal bias. These issues could 
largely affect the empirical results of tourism website evaluation. In 
short, the tourism industry at present does not have, but urgently 
needs, commonly agreed-upon website evaluation techniques that 
are repeatable and measurable and have a good potential for long¬ 
term use. Although these repeatable and measurable techniques 
are unlikely applicable to all sectors in tourism, such techniques can 
be used to formulate plans for a specific business. 

5. Conclusions and future work 

The trend in the structure of evaluation instruments for tourism 
website evaluation is presented in Fig. 2. The following paragraphs 
explain the characteristics of these instruments and the future 
trend. 

In the initial research in this field, researchers generally chose 
qualitative methods to examine the views of certain groups, 
including consumers, suppliers, and academic researchers, on the 
specific features or functions of tourism-related websites. A major 
limitation of these early studies is the non-generalizability of the 
findings, as only a limited number of people participated in the 
experiments. Also, although involving academic researchers in the 
process can ensure the validity and reliability of the instruments, 
researchers and their research assistants have limited experience in 
online purchasing. With time, however, website evaluation studies 
have evolved to include the application of quantitative methods on 
the basis of input from consumers and practitioners. Although the 
concern about the generalizability of the results remains, the 
findings of quantitative research appear to be more applicable to 
larger market segments than do those of qualitative research. 
Presently, many researchers are integrating quantitative and qual¬ 
itative approaches in their website evaluation studies. 

To date, there has been only limited research into tourism 
website evaluation, and that using the previously mentioned 
approaches has achieved only a moderate degree of success. These 
approaches may not be adequate to measure what motivates users 
to browse and make purchases on travel websites. In other words, 
consumers and practitioners do not have sufficient insight into how 
website performance may be accurately measured. The existing 
tourism literature simply does not have any commonly agreed- 
upon standards or techniques for website evaluation. Thus, a future 
research direction would be to investigate the feasibility of devel¬ 
oping sector (or subsectorj-specific standards for tourism website 
evaluation. Another direction for evaluation model/instrument 
research would be the development of new techniques for con¬ 
ducting research that focuses on the needs of consumers and 
practitioners, as the behavior of these users may change after they 
have used the Internet for some time. 

Specifically, theories, algorithms, and models from other disci¬ 
plines such as psychology, human-computer interaction, and engi¬ 
neering could, and should, be incorporated into the tourism website 
evaluation process. After all, tourism practitioners set up their web¬ 
sites, representing public places in the virtual environment, for 
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Fig. 2. A structured trend in website evaluation approaches in tourism. 
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information dissemination and sales. It is of paramount importance to 
be aware of the different range of human possibilities of intentions 
behind the websites. In addition, seeking the views of industrial 
practitioners and consumers remains important, as these groups are 
the ultimate suppliers and users of tourism websites. 

This paper has analyzed prior studies that use diverse meth¬ 
odological approaches to tourism website evaluation. The findings 
are expected to benefit practitioners and researchers by helping 
them better understand what has been achieved thus far. For 
industry practitioners, the findings can serve as an index to help 
them identify the strengths and weaknesses of their websites and 
possibly determine their relative position in the marketplace. For 
academic researchers, this study is expected to offer insights into 
research gaps, which indicate areas for further research on tourism 
website evaluation in general and methods of evaluation in 
particular. As previously stated, the development of integrated 
approaches that incorporate different algorithms and theories from 
other disciplines, such as psychology and computer science, into 
the evaluation process would, and should, be a future research 
direction. 

As not all databases were searched, it is possible that a few 
related published articles could have been missed in the data 
collection stage. In addition, this study focused on analyzing web¬ 
site evaluation techniques, and details of the attributes in different 
evaluation frameworks were excluded. It is likely that different 
tourism sectors would use different evaluation instruments. For 
this reason, it would be worthwhile for future research to analyze 
the attributes and dimensions of such instruments for individual 
industrial sectors and the tourism industry as a whole. As tourism 
website evaluation is a newly emerged area of research, more 
nontrivial trial-and-error efforts are needed before generalized 
conclusions can be drawn. After all, tourism websites are primarily 
built for consumers and industrial practitioners, and users need to 
be engaged in the actual evaluation process. 
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